Method and apparatus for encoding a digital video signal

ABSTRACT

The present invention relates to a method and an encoder for encoding a digital video sequence, said digital video sequence comprising some sets of images including a disparity map, said disparity map being used to reconstruct one image of a set of images from another image of said set of images. The method is characterized in that it comprises the steps of:—encoding a type of a disparity map to be used for the reconstruction of an image, and—encoding the disparity map. Use: encoder in a video communication system

The present invention relates to a method of encoding a digital videosequence, said digital video sequence comprising some sets of imagesincluding disparity maps, a disparity map being used to reconstruct oneimage of a set of images from a reference image of said set of images.The invention also relates to an encoder, said encoder implementing saidmethod.

Such a method may be used in, for example, a video communication systemfor 3D video applications within MPEG standards.

BACKGROUND OF THE INVENTION

A video communication system typically comprises a transmitter with anencoder and a receiver with a decoder. Such a system receives an inputdigital video sequence, encodes said sequence via the encoder, transmitsthe encoded sequence to the receiver, then decodes the transmittedsequence via the decoder, resulting in an output digital video sequence,which is the reconstructed sequence of the input digital video sequence.The receiver then displays said output digital video sequence. A 3Ddigital video sequence comprises some sets of images with objects,usually one first set of texture images along with another set of imagescalled disparity images or disparity maps. An image comprises somepixels.

Each image of the digital video signal is encoded along differentgeneral coding schemes, which have already been proposed within thescope of MPEG. For example, the MPEG2 standard referenced “Draftamendment N^(o) 3 to 13818-2 Multi-view profile-JTC1/SC29/WG11N1088”edited by ISO/IEC in November 1995 during the MPEG Meeting of Dallas(Tex.), has set the basis for the encoding of different views of a samevideo sequence. The main principle is not only, as in most traditionalvideo coding schemes, to use temporal and spatial redundancies withinone video sequence, but also to use redundancies between the differentpoints of view within a video sequence, wherein each point of view is animage, a left image and a right image catch, respectively, by a leftcamera and a right camera, for example. As objects of a video sequenceseen from two slightly different points of view do not differ very much,it is possible to predict a large part of points of view from referencepoints of view by virtue of prediction vectors also called disparityvectors.

Since it is always possible to have disparity vectors that are all alongthe same direction, it often supposed that there are only horizontaldisparity vectors. In this case, a disparity vector is defined by asingle value, called disparity value. The disparity map is an image inwhich a disparity value is assigned to every pixel.

These disparity values are encoded by the encoder and transmitted to thedecoder. A reference image is also sent to the decoder, for example theleft one. Said decoder will use, amongst other parameters, the disparityvalues to reconstruct the right image from the reference image.

There are various encoding schemes well known to the person skilled inthe art, like like DCT based, lossless run-length coding or mesh-basedschemes, which can be used to encode an image. In all these encodingschemes, the disparity values are usually encoded on n-integer values,often on 8-bit data representing 256 gray levels.

One inconvenience of these encoding schemes is that, at the receiverside, one does not know exactly how to translate the disparity map of atexture image solely from these gray-level data.

Indeed, depending on a video sequence content, the disparity map of atexture image can change dramatically and hence the translation.

If the video sequence contains only objects filmed at a very closedistance, disparity may need to be quite accurate, with sub-pixelaccuracy. On the contrary, if the camera focuses on pretty far objects,sub-pixel accuracy might be of no interest, whereas there might be somevery large values of disparity. Finally, there might be a mixedsituation, with different regions of interests within the scene and aneed of non-linear varying set of values of disparity.

Therefore, because of this problem of translation of the disparity mapof the prior art, at the receiver side, there is often a manual tuningof the 3D display in order to:

-   -   view correctly in 3D the reconstructed video sequence, so that a        reconstructed image is equal to, or has few distortions compared        to the original one, and/or    -   to view correctly in 3D a second 3D video sequence after a        previous 3D video sequence, sent by 2 different broadcasters,        for example, if these two video sequences have totally different        disparity values assigned to them.

If the manual tuning has to be done very often, it will cause discomfortfor a viewer of a 3D video sequence.

OBJECT AND SUMMARY OF THE INVENTION

Accordingly, it is an object of the invention to provide a method and anencoder for encoding a digital video sequence, said digital videosequence comprising some sets of images including disparity maps, adisparity map being used to reconstruct one image of a set of imagesfrom a reference image of said set of images, which allow a precisetranslation of the disparity map.

To this end, the method comprises the steps of:

-   -   encoding a type of the disparity map to be used for the        reconstruction of an image, and    -   encoding the disparity map.

In addition, there is provided an encoder comprising first encodingmeans adapted to encode a type of the disparity map to be used for thereconstruction of an image, and second encoding means for encoding thedisparity map.

As we will see in detail, by encoding the type of the disparity map, andmore precisely by encoding the way to compute the disparity values fromthe 8 bits of gray levels, the disparity map of 3D video sequences isefficiently represented and the processing of the disparity map on theDisplay side of the video chain is made automatically.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional objects, features and advantages of the invention will becomeapparent upon reading the following detailed description and uponreference to the accompanying drawings in which:

FIG. 1 illustrates a video communication system comprising an encoderand a decoder according to the invention, and

FIG. 2 is schematic diagram of the encoding method performed by theencoder of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, functions or constructions well known tothe person skilled in the art are not described in detail because theywould obscure the invention in unnecessary detail.

The present invention relates to a method for encoding a digital videosequence, said digital video sequence comprising some sets of images,usually one first set of texture images along with another set of imagescalled disparity images or disparity maps. A disparity map is used toreconstruct one image of a set of texture images from a reference imageof said set of texture images.

Such a method may be used within a video communication system SYS for 3Dvideo applications in MPEG2 or MPEG4, wherein said video communicationsystem comprises a transmitter TRANS, a transmission medium CH and areceiver RECEIV. Said transmitter TRANS and said receiver RECEIVcomprise an encoder ENC and a decoder DEC, respectively.

In order to transmit efficiently some video sequences through thetransmission medium CH, said encoder ENC applies an encoding on a videosequence, then the encoding video sequence is sent to a decoder DEC,which decodes said sequence. Finally the receiver RECEIV displays saidvideo sequence.

A 3D video sequence comprises some sets of images with objects, whereinan image is represented by a plurality of pixels.

One object of a video sequence seen from two slightly different pointsof view does not differ very much. Therefore, a large part of points ofview is predicted from reference points of view by virtue of predictionvectors, also called disparity vectors.

Since it is always possible to have disparity vectors that are all alongthe same direction—by rectification of the original stereo pairaccording to epipolar constraints, for example—it can be supposed thatthere are only horizontal disparity vectors (the common case of a“parallel stereo setting” of video cameras). In this case, a disparityvector is defined by a single value, called disparity value. In theremainder of the description, a disparity vector will be referred to asdisparity value. Of course, this should in no way be restrictive. Thedisparity map is an image in which a disparity value is assigned toevery pixel.

These disparity values allows definition of the shifting of a pixel ofan object between a reference image and another image, at a time t, forexample when said two images represent two different points of view of asame scene of the video sequence. The two points of view of a scene areissued by two cameras placed at different spots.

In order to be efficiently coded by compression algorithms, thedisparity values are represented by n-integer values, often on 8-bitdata representing 256 gray levels. The main issue is that thetranslation between the encoded n-integer values and the disparityvalues may be of different types.

The disparity map also relates to the depth of the objects of an image.Roughly, in most classic representations of 3D images, the more anobject is far in a reference image (large depth), the less the movementsof said object will be apparent in the reconstructed image. On thecontrary, the more an object is near in the reference image, the morethe movements of said object will be apparent in the reconstructedimage.

In order to reduce the information that is transmitted via thetransmission medium, redundancies between points of view are used. Thus,as objects seen from two different points of view do not differ verymuch, it is possible to predict one point of view from the other one.One point of view, the reference one, will be encoded and sent via thetransmission medium CH to the receiver RECEIV. Said receiver RECEIV willdecode it, reconstruct the original reference point of view and deducethe other point of view from the reference one thanks to the disparityvectors or values assigned to said reference point of view.

The encoder ENC comprises first encoding means adapted to encode a typeof a disparity map to be used for the reconstruction of an image, andsecond encoding means for encoding the disparity map.

The encoding of a video sequence is done as follows and is illustratedin FIG. 2.

In a first step 1), the type of the disparity map is encoded, whereinthe type represents the way the disparity values are to be translated,i.e. computed. In a non-limitative embodiment, a flag C1 encodes saidtype of disparity map. In a first variant mode of said embodiment, saidflag C1 is set for each image within a video sequence. In a secondvariant mode of said embodiment, said flag C1 is set for a group ofimages, for example in the header of a group of images, said headerbeing defined in the standard MPEG2 referenced “ISO/IEC 13818-2:2000Information technology—Generic coding of moving pictures and associatedaudio information: Video”.

This group of images, also referred to as GOP “Group Of Pictures”, wouldhave the particularity of having a same disparity map representation,i.e. the disparity values are computed in the same manner. The type flagcan be coded on 3 bits, for example, to represent the disparity map. Itmay also have a variable length.

The following non-limitative representations can be applied for thedisparity map: affine, logarithmic, polynomial, piecewise planar.

For example, in case of an affine representation, the disparity value iscomputed with the following formula.

Disparity value=(N-integer−Shift)/Dynaimic, wherein N_integer representsthe 256 gray levels coded on 8 bits, Shift represents the 3Dstereoscopic character of an image in relation to a user of the videosystem like a television (3D image giving the impression of being “in”or “out” of the screen), coded on 8 bits, and liamic represents thedepth of the objects amongst them, coded on 4 bits.

In a second step 2), if the representation of the disparity maprepresentation needs some parameters, these parameters are also encoded.

For example, in the case of the affine representation, the shift and thedynamic values are two parameters P1 and P2 that are encoded.

In a third and last step 3), the disparity map, i.e. the gray levels, isencoded with general coding methods like DCT, lossless method, meshmethod . . . .

Preferably, the flag(s) C1 and the associated parameters P1, P2 . . .are put before the encoded disparity map. They are not necessarilytransmitted just before the disparity map.

Note that a flag, and as the case may be its associated parameters P1,P2 . . . , are transmitted with the associated image or group of images.

At the decoder DEC side, the knowledge of the type flag will tell saiddecoder if it has to wait for additional parameters or not.

Thus, one advantage of the present invention is to tell the decoder, andtherefore the receiver, how to use exactly the disparity representationon an image to reconstruct an image of a set of texture images fromanother one.

The use of a flag allows simple definition of the type of a disparitymap. Moreover, it does not use too much memory, contrary to the use of atable, which would attribute to each value of the gray levels anexplanation about how to move a pixel, for example.

Such a table has also the inconvenience of being transmitted each timethe disparity map representation changes, that is to say, a lot of bitshave to be transmitted.

Another advantage of the present invention is that it improves thereconstruction of a point of view on the basis of a reference point ofview and the associated disparity map. Indeed, with the flag C1 and, asthe case may be, with the parameters, the reconstruction of thereconstructed point of view is more precise and thus, the reconstructedpoint of view better fits the original point of view. The usage of theflag(s) to explain how the disparity map shall be interpreted allowsconsistent 3D effects to the viewer, whatever translation function wasoriginally used to encode the disparity values.

Finally, a third advantage of the present invention is that, when itcomes to the reconstruction of one view on the basis of a reference viewand the associated disparity map, we have to fill the holescorresponding to parts of the reconstructed view that are not viewed inthe reference view. The width of these holes depends on the dynamic ofdisparity an thus on the representation of the disparity map. If onewants to build an enhancement layer of images devoted to the filling ofthe holes in the reconstructed views, precise references to the way tocompute the disparity values is now available.

It is to be understood that the present invention is not limited to theaforementioned embodiments and variations and modifications may be madewithout departing from the spirit and scope of the invention as definedin the appended claims. In this respect, the following closing remarksare made.

It is to be understood that the present invention is not limited to theaforementioned 3D video application. It can be used within anyapplication using a system for processing a signal where said signal ischaracterized by gray levels such as a heating signal.

It is to be understood that the method according to the presentinvention is not limited to the aforementioned implementation.

There are numerous ways of implementing functions of the methodaccording to the invention by means of items of hardware or software, orboth, provided that a single item of hardware or software can carry outseveral functions. It does not exclude that an assembly of items ofhardware or software or both carry out a function, thus forming a singlefunction without modifying the method for processing the video signal inaccordance with the invention.

Said hardware or software items can be implemented in several manners,such as by means of wired electronic circuits or by means of anintegrated circuit that is suitably programmed, respectively. Theintegrated circuit may be contained in a computer or in an encoder. Inthe second case, the encoder comprises first encoding means adapted toencode a type of a disparity map to be used for the reconstruction of animage, and second encoding means for encoding the disparity map, asdescribed previously, said means being hardware or software items asstated above.

The integrated circuit comprises a set of instructions. Thus, said setof instructions contained, for example, in a computer programming memoryor in an encoder memory may cause the computer or the encoder to carryout the different steps of the decoding method.

The set of instructions may be loaded into the programming memory byreading a data carrier in the form of a computer-readable medium, suchas, for example, a disc. A service provider can also make the set ofinstructions available via a communication network such as, for example,the Internet.

Any reference sign in the following claims should not be construed aslimiting the claim. It will be obvious that the use of the verb “tocomprise” and its conjugations does not exclude the presence of anyother steps or elements besides those defined in any claim. The article“a” or “an” preceding an element or step does not exclude the presenceof a plurality of such elements or steps.

1. A method of encoding a digital video sequence for use within a videocommunication system, said digital video sequence comprising some setsof images including a disparity map comprising an image in which adisparity value is assigned to every pixel, said disparity map beingused to reconstruct one image of a set of images from a reference imageof said set of images, characterized in that the method comprises thesteps of: encoding with a first encoding means a type of the disparitymap to be used for the reconstruction of an image, wherein the type (i)represents the way that disparity values of the disparity map are to betranslated by a decoder within the video communication system, and (ii)explains to the decoder how to use exactly the disparity map on thereference image to reconstruct one image of the set of images fromanother one; and encoding with a second encoding means the disparitymap.
 2. The method of processing a digital video sequence as claimed inclaim 1, characterized in that the encoding of the type of the disparitymap is done by means of a flag, wherein use of the flag allows simpledefinition of the type of disparity map.
 3. The method of processing adigital video sequence as claimed in claim 1, characterized in that theencoding of the type of the disparity map is followed by a set ofencoded parameters.
 4. A computer-readable medium having encoded thereona computer program for execution by an encoder, said computer programcomprising a set of instructions, which, when loaded into said encoder,causes the encoder to carry out the method claimed in any one of claims1 to
 3. 5. A computer-readable medium having encoded thereon a computerprogram for execution by a computer, said computer program comprising aset of instructions, which, when loaded into said computer, causes thecomputer to carry out the method claimed in any one of claims 1 to
 3. 6.An encoder for encoding a digital video sequence, said digital videosequence comprising some sets of images including a disparity mapcomprising an image in which a disparity value is assigned to everypixel, said disparity map being used to reconstruct one image of a setof images from a reference image of said set of images, characterized inthat the encoder comprises: first encoding means for encoding a type ofthe disparity map to be used for the reconstruction of an image, whereinthe type (i) represents the way that disparity values of the disparitymap are to be translated by a decoder within a video communicationsystem, and (ii) explains to the decoder how to use exactly thedisparity map on the reference image to reconstruct one image of the setof images from another one; and second encoding means for encoding thedisparity map.
 7. A video communication system, which is able to receivea digital video sequence, said video communication system comprising anencoder as claimed in claim 6 for encoding said video signal, atransmission channel for transmitting the encoded video signal, and adecoder for decoding said encoded video signal.